{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction to Quantitative Finance\n",
    "\n",
    "Copyright (c) 2019 Python Charmers Pty Ltd, Australia, <https://pythoncharmers.com>. All rights reserved.\n",
    "\n",
    "<img src=\"img/python_charmers_logo.png\" width=\"300\" alt=\"Python Charmers Logo\">\n",
    "\n",
    "Published under the Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0) license. See `LICENSE.md` for details.\n",
    "\n",
    "Sponsored by Tibra Global Services, <https://tibra.com>\n",
    "\n",
    "<img src=\"img/tibra_logo.png\" width=\"300\" alt=\"Tibra Logo\">\n",
    "\n",
    "\n",
    "## Module 1.1: Distributions and Random Processes\n",
    "\n",
    "### 1.1.4: Normality Tests\n",
    "\n",
    "If you are analysing data on the assumption it is normally distributed, you should test that assumption first. Properties of normal distributions do not necessarily apply to data that has a different underlying distribution. As an example, an ANOVA test assumes normality in your data, and the results of an ANOVA are not valid if the data comes from some other source.\n",
    "\n",
    "There are a number of normality tests that provide a way for us to determine if it is likely that the data comes from a normal distribution.\n",
    "\n",
    "One method of testing for normality is to compute the skew and kurtosis of the data. A standard normal distribution has a skew of 0 and a kurtosis of 3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%run setup.ipy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "aapl = pd.read_pickle(\"data/AAPL.pkl\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "aapl['Gain'] = aapl['Adj Close'].diff()\n",
    "aapl.dropna(inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.49966243998511045"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stats.skew(aapl['Gain'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "20.448973653158085"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stats.kurtosis(aapl['Gain'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The AAPL stock price increases do not appear to have a normal distribution based on this data. Let's have a look at the histogram again:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "  #altair-viz-553c04fd4c9c4191ac8cb16be0dbee8b.vega-embed {\n",
       "    width: 100%;\n",
       "    display: flex;\n",
       "  }\n",
       "\n",
       "  #altair-viz-553c04fd4c9c4191ac8cb16be0dbee8b.vega-embed details,\n",
       "  #altair-viz-553c04fd4c9c4191ac8cb16be0dbee8b.vega-embed details summary {\n",
       "    position: relative;\n",
       "  }\n",
       "</style>\n",
       "<div id=\"altair-viz-553c04fd4c9c4191ac8cb16be0dbee8b\"></div>\n",
       "<script type=\"text/javascript\">\n",
       "  var VEGA_DEBUG = (typeof VEGA_DEBUG == \"undefined\") ? {} : VEGA_DEBUG;\n",
       "  (function(spec, embedOpt){\n",
       "    let outputDiv = document.currentScript.previousElementSibling;\n",
       "    if (outputDiv.id !== \"altair-viz-553c04fd4c9c4191ac8cb16be0dbee8b\") {\n",
       "      outputDiv = document.getElementById(\"altair-viz-553c04fd4c9c4191ac8cb16be0dbee8b\");\n",
       "    }\n",
       "    const paths = {\n",
       "      \"vega\": \"https://cdn.jsdelivr.net/npm/vega@5?noext\",\n",
       "      \"vega-lib\": \"https://cdn.jsdelivr.net/npm/vega-lib?noext\",\n",
       "      \"vega-lite\": \"https://cdn.jsdelivr.net/npm/vega-lite@5.16.3?noext\",\n",
       "      \"vega-embed\": \"https://cdn.jsdelivr.net/npm/vega-embed@6?noext\",\n",
       "    };\n",
       "\n",
       "    function maybeLoadScript(lib, version) {\n",
       "      var key = `${lib.replace(\"-\", \"\")}_version`;\n",
       "      return (VEGA_DEBUG[key] == version) ?\n",
       "        Promise.resolve(paths[lib]) :\n",
       "        new Promise(function(resolve, reject) {\n",
       "          var s = document.createElement('script');\n",
       "          document.getElementsByTagName(\"head\")[0].appendChild(s);\n",
       "          s.async = true;\n",
       "          s.onload = () => {\n",
       "            VEGA_DEBUG[key] = version;\n",
       "            return resolve(paths[lib]);\n",
       "          };\n",
       "          s.onerror = () => reject(`Error loading script: ${paths[lib]}`);\n",
       "          s.src = paths[lib];\n",
       "        });\n",
       "    }\n",
       "\n",
       "    function showError(err) {\n",
       "      outputDiv.innerHTML = `<div class=\"error\" style=\"color:red;\">${err}</div>`;\n",
       "      throw err;\n",
       "    }\n",
       "\n",
       "    function displayChart(vegaEmbed) {\n",
       "      vegaEmbed(outputDiv, spec, embedOpt)\n",
       "        .catch(err => showError(`Javascript Error: ${err.message}<br>This usually means there's a typo in your chart specification. See the javascript console for the full traceback.`));\n",
       "    }\n",
       "\n",
       "    if(typeof define === \"function\" && define.amd) {\n",
       "      requirejs.config({paths});\n",
       "      require([\"vega-embed\"], displayChart, err => showError(`Error loading script: ${err.message}`));\n",
       "    } else {\n",
       "      maybeLoadScript(\"vega\", \"5\")\n",
       "        .then(() => maybeLoadScript(\"vega-lite\", \"5.16.3\"))\n",
       "        .then(() => maybeLoadScript(\"vega-embed\", \"6\"))\n",
       "        .catch(showError)\n",
       "        .then(() => displayChart(vegaEmbed));\n",
       "    }\n",
       "  })({\"config\": {\"view\": {\"continuousWidth\": 300, \"continuousHeight\": 300}}, \"data\": {\"url\": \"altair-data-c00380db16792b66753e6de91a6b15db.json\", \"format\": {\"type\": \"json\"}}, \"mark\": {\"type\": \"bar\"}, \"encoding\": {\"x\": {\"bin\": {\"maxbins\": 100}, \"field\": \"Gain\", \"type\": \"quantitative\"}, \"y\": {\"aggregate\": \"count\", \"type\": \"quantitative\"}}, \"$schema\": \"https://vega.github.io/schema/vega-lite/v5.16.3.json\"}, {\"mode\": \"vega-lite\"});\n",
       "</script>"
      ],
      "text/plain": [
       "alt.Chart(...)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "alt.Chart(aapl).mark_bar().encode(\n",
    "        alt.X(\"Gain\", bin=alt.Bin(maxbins=100)),\n",
    "        y='count()',\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A high Kurtosis is obtained from very \"sharp\" peaks such as this one. The skew is not that high, but is positive, indicating a right-leaning distribution.\n",
    "\n",
    "More objective tests are available in the `scipy.stats` package. For instance, the Shapiro-Wilk test is commonly used and is a good test for small to medium datasets, with up to a few thousand data points."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "statistic, p = stats.shapiro(aapl['Gain'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.0"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "p  # Compare the p value to your acceptable alpha value to determine if the null hypothesis can be rejected."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The data does not look like it was drawn from a normal distribution\n"
     ]
    }
   ],
   "source": [
    "if p > 0.05:\n",
    "    print(\"The data looks like it was drawn from a normal distribution\")\n",
    "else:\n",
    "    print(\"The data does not look like it was drawn from a normal distribution\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What is a p-value?\n",
    "\n",
    "The p-value above is a commonly used term to describe the probability of your test being true.\n",
    "\n",
    "As it is a probability, it has a value between 0 and 1. Values near 0 indicate that your test is \"not likely to be true\" and values near 1 indicate that your test is likely to be true. Often, we apply a threshold, and if our p value is greater than that threshold, we accept the outcome as \"likely enough, and we continue as if it were true\", that is, we accept the outcome of the test as a \"positive\".\n",
    "\n",
    "It is very common to use a threshold of 0.05 when performing a test. That is, if our test has a greater than 95% chance of being true, we accept it as such. While this is an adequate rule of thumb, it is not a one-size-fits-all solution to the problem of choosing a p value threshold.\n",
    "\n",
    "Where this is normally seen in classical statistics is with a Null, and Alternative hypothesis. We will delve into these deeper later, but as this is used above, the null hypothesis is our \"nothing is surprising\" hypothesis, and the alternative is \"there is something interesting here\". For the Shapiro-Wilk used above, the hypothesis are:\n",
    "\n",
    "* $H_0$ (the Null hypothesis): The data is drawn from a normal distribution\n",
    "* $H_A$ (the Alternative hypothesis): The data was not drawn from a normal distribution\n",
    "\n",
    "Here we have mutually exclusive tests. If we get a value of $a$ for our Null hypothesis, then the probability of our Alternative being true is $1-a$. Statisticians are a pessimistic bunch, so require a very high threshold before we reject the Null hypothesis. This is akin to requiring a very high amount of evidence to reject it. Therefore, to reject the Null hypothesis, to indicate something else is going on here, we require the p value to be less than 0.05, i.e. for there to be a greater than 95% chance the Alternative hypothesis is true.\n",
    "\n",
    "This might seem like a high standard to meet, but humans often see patterns in data that are not there. We use statistics to test these patterns and ensure we don't fall afoul of this over confident pattern matching.\n",
    "\n",
    "Before you decide to run a new statistical test, you should see first what the p value would tell you. Often the language is \"accept the null hypothesis\" or \"fail to accept the null hypothesis\". This will tell you how to use the test."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We could see from the Kurtosis that this dataset above wasn't normal. Let's look at a different set of data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "heights = np.array([\n",
    "    205.61624376, 155.80577135, 202.09636984, 159.19312848,\n",
    "    160.0263383 , 147.44200373, 160.96891569, 160.76304892,\n",
    "    167.59165377, 164.31571823, 151.11269914, 176.43856129,\n",
    "    176.88435091, 138.04177187, 183.87507305, 162.81488426,\n",
    "    167.96767641, 144.68437342, 180.88771461, 179.18997091,\n",
    "    189.81672505, 163.68662119, 175.70135072, 167.32793289,\n",
    "    163.72509862, 207.93257342, 177.41722601, 167.28154916,\n",
    "    170.26294662, 187.01142671, 178.3108478 , 168.8711774 ,\n",
    "    202.77222671, 138.55043572, 187.10284379, 155.13494037,\n",
    "    175.24219374, 188.54739561, 191.42024196, 174.34537673,\n",
    "    158.36285104, 183.17014557, 166.36310929, 185.3415384 ,\n",
    "    163.87673308, 173.70401469, 168.78499868, 167.39762991,\n",
    "    166.89193943, 191.04035344, 148.02108024, 140.82772936,\n",
    "    168.85378921, 142.13536543, 189.77084606, 173.7849811 ,\n",
    "    157.61303804, 171.62493617, 173.30529631, 162.92083214,\n",
    "    169.52974326, 142.01039665, 176.01691215, 170.32439763,\n",
    "    172.64616031, 158.35076247, 185.96332979, 176.6176222 ,\n",
    "    204.68516079, 161.43591954, 172.42384543, 179.36900257,\n",
    "    170.01353653, 194.40269002, 139.96802012, 156.47281846,\n",
    "    210.21895193, 153.30508193, 157.10282665, 200.07040619,\n",
    "    174.69616438, 168.97403285, 188.9396949 , 156.19358617,\n",
    "    179.56494356, 175.04014032, 164.1384659 , 167.90219562,\n",
    "    184.80752625, 143.56580744, 169.80537836, 186.5894398 ,\n",
    "    166.39251657, 165.65510886, 195.49137372, 152.21650272,\n",
    "    163.14001055, 170.27382512, 147.63901378, 190.32910286])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "statistic, p = stats.shapiro(heights)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The data looks like it was drawn from a normal distribution\n",
      "p=0.278\n"
     ]
    }
   ],
   "source": [
    "if p > 0.05:\n",
    "    print(\"The data looks like it was drawn from a normal distribution\")\n",
    "    print(\"p={:.3f}\".format(p))\n",
    "else:\n",
    "    print(\"The data does not look like it was drawn from a normal distribution\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise\n",
    "\n",
    "Two other commonly used tests for normality are available in `scipy.stats`. They are `stats.normaltest` and `stats.kstest`. Review the help and references for these functions, and run them on the `heights` data. What are the strengths and weaknesses of each test?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.6994130645220737 0.0\n",
      "From Stack overflow, the Kolmogorov-Smirnov is for a completely specified distribution, while the Shapiro-Wilk is for normality, with unspecified mean and variance.\n"
     ]
    }
   ],
   "source": [
    "statistic_chi, p_c = stats.normaltest(heights) #chi squared test statistic\n",
    "statistic_k, p_k = stats.kstest(heights,cdf = 'norm') #another one?\n",
    "print(str(p_c), str(p_k))\n",
    "\n",
    "print(\"From Stack overflow, the Kolmogorov-Smirnov is for a completely specified distribution, while the Shapiro-Wilk is for normality, with unspecified mean and variance.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*For solutions, see `solutions/scipy_normal_tests.py`*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Statsmodels\n",
    "\n",
    "We will now perform a normality test using the `statsmodels` package. This package allows for higher level statistics than the `scipy` module we have been using. We will be using `statsmodels` for much of the ordinary least squares computation in future modules."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "import statsmodels.api as sm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "statistic, p_value = sm.stats.diagnostic.kstest_normal(heights)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The data looks like it was drawn from a normal distribution\n",
      "p=0.395\n"
     ]
    }
   ],
   "source": [
    "if p_value > 0.05:\n",
    "    print(\"The data looks like it was drawn from a normal distribution\")\n",
    "    print(\"p={:.3f}\".format(p_value))\n",
    "else:\n",
    "    print(\"The data does not look like it was drawn from a normal distribution\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise\n",
    "\n",
    "Review the documentation for `statsmodels` at https://www.statsmodels.org and run the Jarque-Bera test for normality on this data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.6714923453511482\n"
     ]
    }
   ],
   "source": [
    "from statsmodels.stats import stattools\n",
    "\n",
    "jbstat, pvalue, skew, kurtosis = stattools.jarque_bera(heights)\n",
    "print(pvalue)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*For solutions, see `solutions/jarque_bera.py`*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Handling conflicts\n",
    "\n",
    "There are many different normality tests. If you get the same result for all the tests (i.e. multiple tests suggest normal data), then you can be reasonably sure the data does come from a normal distribution.\n",
    "\n",
    "If you get conflicting result, the results are not quite so clear. In a conflicting case, it would be unlikely that the results will be wildly different. Instead, you are likely to get a few slightly \"above the line\" and a few slightly \"below the line\". Depending on the use case, you can interpret a single \"is normal\" result as being good enough. Much of the later analysis you can do will be fine for \"normal-like\" data, rather than strictly normal data.\n",
    "\n",
    "If you do have a very sensitive application that requires a great degree in confidence in your normality test, research further the assumptions behind different normality tests and see which are most applicable to your application. The SciPy and Statsmodels documentation contain references for each of the normality tests.\n",
    "\n",
    "A major property is the number of samples in your dataset. Some tests work better with more samples, and some work better with fewer. We will investigate this in the last exercise for this module."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise\n",
    "\n",
    "We are going to investigate the relationship that sample size has with the results of a normality test. We want to test the likelihood a normality test will reject the normality hypothesis for a dataset that *actually is generated from a normal distribution*, as the sample size increases.\n",
    "\n",
    "Write a script that:\n",
    "\n",
    "1. Creates a normal distribution\n",
    "2. Randomly samples N data points from that distribution\n",
    "3. Checks for normality against four different normality tests\n",
    "4. Repeats steps 1-3 a large number of times, and with varying N\n",
    "5. Plot the likelihood each test fails for a given sample size.\n",
    "\n",
    "Below is a snippet of code that runs 20 tests against one sample of data, and determines if the test determines it is normal or not. For an alpha value of 0.05, you would expect about 1 of the tests to fail on average.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "def normality_script(sample_size, test_type, repetitions = 100):\n",
    "    distribution = stats.norm()\n",
    "    data = distribution.rvs(sample_size)\n",
    "    \n",
    "    passed = 0\n",
    "    failed = 0\n",
    "    for i in range(repetitions):\n",
    "        distribution = stats.norm()\n",
    "        data = distribution.rvs(sample_size)\n",
    "    \n",
    "        if test_type == \"sw\":\n",
    "            statistic, p = stats.shapiro(data)\n",
    "        elif test_type == \"cs\":\n",
    "            statistic, p = stats.normaltest(data)\n",
    "        elif test_type == \"ks\":\n",
    "            statistic, p = stats.kstest(data, cdf = 'norm')\n",
    "        elif test_type == \"jb\":\n",
    "            statistic, p, skew, kurtosis = stattools.jarque_bera(data)\n",
    "        \n",
    "        if p > 0.05:\n",
    "            passed += 1\n",
    "        else:\n",
    "            failed += 1\n",
    "        \n",
    "    return failed/(passed+failed)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\Python311\\Lib\\site-packages\\scipy\\stats\\_stats_py.py:1736: UserWarning: kurtosistest only valid for n>=20 ... continuing anyway, n=10\n",
      "  warnings.warn(\"kurtosistest only valid for n>=20 ... continuing \"\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "  #altair-viz-b86b65659a60406182e466216b50c792.vega-embed {\n",
       "    width: 100%;\n",
       "    display: flex;\n",
       "  }\n",
       "\n",
       "  #altair-viz-b86b65659a60406182e466216b50c792.vega-embed details,\n",
       "  #altair-viz-b86b65659a60406182e466216b50c792.vega-embed details summary {\n",
       "    position: relative;\n",
       "  }\n",
       "</style>\n",
       "<div id=\"altair-viz-b86b65659a60406182e466216b50c792\"></div>\n",
       "<script type=\"text/javascript\">\n",
       "  var VEGA_DEBUG = (typeof VEGA_DEBUG == \"undefined\") ? {} : VEGA_DEBUG;\n",
       "  (function(spec, embedOpt){\n",
       "    let outputDiv = document.currentScript.previousElementSibling;\n",
       "    if (outputDiv.id !== \"altair-viz-b86b65659a60406182e466216b50c792\") {\n",
       "      outputDiv = document.getElementById(\"altair-viz-b86b65659a60406182e466216b50c792\");\n",
       "    }\n",
       "    const paths = {\n",
       "      \"vega\": \"https://cdn.jsdelivr.net/npm/vega@5?noext\",\n",
       "      \"vega-lib\": \"https://cdn.jsdelivr.net/npm/vega-lib?noext\",\n",
       "      \"vega-lite\": \"https://cdn.jsdelivr.net/npm/vega-lite@5.16.3?noext\",\n",
       "      \"vega-embed\": \"https://cdn.jsdelivr.net/npm/vega-embed@6?noext\",\n",
       "    };\n",
       "\n",
       "    function maybeLoadScript(lib, version) {\n",
       "      var key = `${lib.replace(\"-\", \"\")}_version`;\n",
       "      return (VEGA_DEBUG[key] == version) ?\n",
       "        Promise.resolve(paths[lib]) :\n",
       "        new Promise(function(resolve, reject) {\n",
       "          var s = document.createElement('script');\n",
       "          document.getElementsByTagName(\"head\")[0].appendChild(s);\n",
       "          s.async = true;\n",
       "          s.onload = () => {\n",
       "            VEGA_DEBUG[key] = version;\n",
       "            return resolve(paths[lib]);\n",
       "          };\n",
       "          s.onerror = () => reject(`Error loading script: ${paths[lib]}`);\n",
       "          s.src = paths[lib];\n",
       "        });\n",
       "    }\n",
       "\n",
       "    function showError(err) {\n",
       "      outputDiv.innerHTML = `<div class=\"error\" style=\"color:red;\">${err}</div>`;\n",
       "      throw err;\n",
       "    }\n",
       "\n",
       "    function displayChart(vegaEmbed) {\n",
       "      vegaEmbed(outputDiv, spec, embedOpt)\n",
       "        .catch(err => showError(`Javascript Error: ${err.message}<br>This usually means there's a typo in your chart specification. See the javascript console for the full traceback.`));\n",
       "    }\n",
       "\n",
       "    if(typeof define === \"function\" && define.amd) {\n",
       "      requirejs.config({paths});\n",
       "      require([\"vega-embed\"], displayChart, err => showError(`Error loading script: ${err.message}`));\n",
       "    } else {\n",
       "      maybeLoadScript(\"vega\", \"5\")\n",
       "        .then(() => maybeLoadScript(\"vega-lite\", \"5.16.3\"))\n",
       "        .then(() => maybeLoadScript(\"vega-embed\", \"6\"))\n",
       "        .catch(showError)\n",
       "        .then(() => displayChart(vegaEmbed));\n",
       "    }\n",
       "  })({\"config\": {\"view\": {\"continuousWidth\": 300, \"continuousHeight\": 300}}, \"data\": {\"name\": \"data-a7eaa51e457e46028771905c13064a4b\"}, \"mark\": {\"type\": \"line\"}, \"encoding\": {\"color\": {\"field\": \"Test\", \"type\": \"nominal\"}, \"x\": {\"field\": \"Sample\", \"type\": \"quantitative\"}, \"y\": {\"field\": \"Failed\", \"type\": \"quantitative\"}}, \"$schema\": \"https://vega.github.io/schema/vega-lite/v5.16.3.json\", \"datasets\": {\"data-a7eaa51e457e46028771905c13064a4b\": [{\"Test\": \"sw\", \"Sample\": 10, \"Failed\": 0.06}, {\"Test\": \"cs\", \"Sample\": 10, \"Failed\": 0.05}, {\"Test\": \"ks\", \"Sample\": 10, \"Failed\": 0.06}, {\"Test\": \"jb\", \"Sample\": 10, \"Failed\": 0.0}, {\"Test\": \"sw\", \"Sample\": 30, \"Failed\": 0.05}, {\"Test\": \"cs\", \"Sample\": 30, \"Failed\": 0.04}, {\"Test\": \"ks\", \"Sample\": 30, \"Failed\": 0.04}, {\"Test\": \"jb\", \"Sample\": 30, \"Failed\": 0.04}, {\"Test\": \"sw\", \"Sample\": 50, \"Failed\": 0.03}, {\"Test\": \"cs\", \"Sample\": 50, \"Failed\": 0.06}, {\"Test\": \"ks\", \"Sample\": 50, \"Failed\": 0.05}, {\"Test\": \"jb\", \"Sample\": 50, \"Failed\": 0.04}, {\"Test\": \"sw\", \"Sample\": 100, \"Failed\": 0.04}, {\"Test\": \"cs\", \"Sample\": 100, \"Failed\": 0.05}, {\"Test\": \"ks\", \"Sample\": 100, \"Failed\": 0.06}, {\"Test\": \"jb\", \"Sample\": 100, \"Failed\": 0.02}, {\"Test\": \"sw\", \"Sample\": 1000, \"Failed\": 0.04}, {\"Test\": \"cs\", \"Sample\": 1000, \"Failed\": 0.05}, {\"Test\": \"ks\", \"Sample\": 1000, \"Failed\": 0.07}, {\"Test\": \"jb\", \"Sample\": 1000, \"Failed\": 0.05}, {\"Test\": \"sw\", \"Sample\": 5000, \"Failed\": 0.04}, {\"Test\": \"cs\", \"Sample\": 5000, \"Failed\": 0.03}, {\"Test\": \"ks\", \"Sample\": 5000, \"Failed\": 0.04}, {\"Test\": \"jb\", \"Sample\": 5000, \"Failed\": 0.05}]}}, {\"mode\": \"vega-lite\"});\n",
       "</script>"
      ],
      "text/plain": [
       "alt.Chart(...)"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "from statsmodels.stats import stattools\n",
    "from scipy import stats\n",
    "import altair as alt\n",
    "import numpy as np\n",
    "\n",
    "sample_sizes = [10, 30, 50, 100, 1000, 5000]\n",
    "#sample_sizes = np.linspace(10, max_sample_size, 20, dtype=np.int)\n",
    "\n",
    "test_types = [\"sw\", \"cs\", \"ks\", \"jb\"]\n",
    "data = []\n",
    "\n",
    "for size in sample_sizes:\n",
    "    for test in test_types:\n",
    "        p_fail = normality_script(size, test)\n",
    "        row = [test,size,p_fail]\n",
    "        data.append(row)\n",
    "        \n",
    "\n",
    "df = pd.DataFrame(data, columns=['Test', 'Sample', 'Failed'])\n",
    "\n",
    "\n",
    "\n",
    "alt.Chart(df).mark_line().encode(\n",
    "    x='Sample',\n",
    "    y='Failed',\n",
    "    color = 'Test'\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "20 passed and 0 failed\n"
     ]
    }
   ],
   "source": [
    "sample_size = 30\n",
    "passed = 0\n",
    "failed = 0\n",
    "for i in range(20):\n",
    "    distribution = stats.norm()\n",
    "    data = distribution.rvs(sample_size)\n",
    "    stat, p = stats.normaltest(data)\n",
    "    if p > 0.05:\n",
    "        passed += 1\n",
    "    else:\n",
    "        failed += 1\n",
    "print(\"{} passed and {} failed\".format(passed, failed))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*For solutions see `solutions/many_normal_tests.py`*"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
